{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "6f4266e5-047d-4e49-a79a-85c99759e1c6",
   "metadata": {},
   "source": [
    "## PII Redactor Example Notebook"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "12bab93e-70ee-4956-821a-2080db9c11a3",
   "metadata": {},
   "source": [
    "\n",
    "**Author**: Pooja Holkar ,\n",
    "**email**:poholkar@in.ibm.com\n",
    "\n",
    "### What is a PII Redactor?\n",
    "\n",
    "A PII (Personally Identifiable Information) Redactor is a tool designed to identify and redact sensitive information in text data. PII includes details that can be used to identify an individual, such as:\n",
    "\n",
    "Names,\n",
    "Email addresses,\n",
    "Phone numbers,\n",
    "Addresses,\n",
    "Financial details (e.g., credit card numbers and crypto addresses).\n",
    "\n",
    "### Overview of the use case\n",
    "In this usecase, the PII Redactor is applied to text extracted from invoices to ensure sensitive customer information is not exposed during processing, sharing, or storage.\n",
    "\n",
    " **Workflow Overview**\n",
    "\n",
    "- **Extracting and Converting Text:** The content of the invoice, originally in PDF format, is processed using the docling2parquet transform to extract the text and convert it into a structured Parquet file, enabling easier handling and downstream processing.\n",
    "\n",
    "- **Redacting Sensitive Information:** The generated Parquet file serves as the input for the dpk_pii_redactor transform. This step scans the invoice data for personally identifiable information (PII) and applies masking techniques to redact any sensitive content, ensuring data privacy and compliance.\n",
    "\n",
    "- **Creating the Final Output:** After the redaction process, a new output Parquet file is generated in **output-redacted** folder, containing the same structured data as the original but with all sensitive details securely masked to prevent unauthorized access or exposure.\n",
    "\n",
    "\n",
    "### Why is PII Redaction Important?\n",
    "\n",
    " **Data Privacy Compliance**: Adheres to regulations like GDPR, HIPAA, or CCPA that mandate safeguarding customer information.\n",
    "\n",
    " **Risk Mitigation**: Prevents unauthorized access to or misuse of sensitive data.\n",
    "\n",
    " **Automation Benefits**: Simplifies and accelerates the process of securing information in large-scale document handling.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "26f701c8",
   "metadata": {},
   "source": [
    "## How to run this notebook\n",
    "\n",
    "If you have python 3.11 or higher on your machine, you can also download the notebook and run it locally using a local python environment setup as follows:\n",
    "\n",
    "```\n",
    "python -m venv venv\n",
    "source venv/bin/activate\n",
    "pip install jupyterlab\n",
    "jupyter lab Run_your_first_PII_redactor_transform.ipynb\n",
    "```\n",
    "\n",
    "For more advanced setup, please see setup [guide](https://github.com/data-prep-kit/data-prep-kit/blob/dev/doc/quick-start/quick-start.md).\n",
    "\n",
    "An earlier version of this notebook was tested successfully on Google Colab. However, continuous changes in the Colab environment could introduce unexpected behavior/breakage. If you wish to try the Colab environment, click on [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/data-prep-kit/data-prep-kit/blob/dev/recipes/PII/Run_your_first_PII_redactor_transform.ipynb).\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "10a3ed52-68aa-45bb-a2d3-1b08e5ad2ab2",
   "metadata": {},
   "source": [
    "### Pre-req: Install data-prep-kit toolkit"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fd7131c9-af67-4ac5-9d3d-a90ab7d11898",
   "metadata": {},
   "outputs": [],
   "source": [
    "#%%capture cap --no-stderr\n",
    "#%pip install \"data-prep-toolkit @ git+https://github.com/touma-I/data-prep-kit-pkg@numpy-dependencies#subdirectory=data-processing-lib\"\n",
    "#%pip install \"data-prep-toolkit-transforms[docling2parquet,pii_redactor] @ git+https://github.com/touma-I/data-prep-kit-pkg@numpy-dependencies#subdirectory=transforms\"\n",
    "%pip install \"data-prep-toolkit-transforms[docling2parquet,pii_redactor]==1.1.5\"\n",
    "%pip install pandas \n",
    "import pandas as pd\n",
    "import os"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2d6f9a32-6ae4-4bf2-b939-70b579f18185",
   "metadata": {},
   "source": [
    "## Step 1: Configuration"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e34249a4-cf6e-4879-a8ed-3092653874a8",
   "metadata": {},
   "source": [
    "### Download Data and set input and output directories\n",
    "#### We will place the downloaded files in the `input_dir`. For our use case, we have used Invoice data file that will undergo processing. The output for each transform run will be generated in separate directories, with directory names following the format `files_<transform_name>`, making it easy to verify the respective transform outputs. This concludes the setup section."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b2186d9f-f09f-4831-be8b-ab04bfe63e41",
   "metadata": {},
   "outputs": [],
   "source": [
    "import urllib.request\n",
    "import shutil\n",
    "shutil.os.makedirs(\"tmp/input\", exist_ok=True)\n",
    "urllib.request.urlretrieve(\"https://raw.githubusercontent.com/data-prep-kit/data-prep-kit/dev/recipes/input-data/PII/Invoice.pdf\", \"tmp/input/Invoice.pdf\")\n",
    "urllib.request.urlretrieve(\"https://raw.githubusercontent.com/data-prep-kit/data-prep-kit/dev/recipes/input-data/PII/Cryptotest.pdf\", \"tmp/input/Cryptotest.pdf\")\n",
    "\n",
    "input_dir = \"tmp/input\"\n",
    "output_dir = \"output\"\n",
    "output_docling2pq_dir = os.path.join (output_dir, 'files_docling2parquet')\n",
    "output_piiredactor_dir = os.path.join (output_dir, 'files_piiredacted')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "348d002f-1458-48c7-987c-ba45de4cb2f2",
   "metadata": {},
   "source": [
    "## Step 2: Invoke Docling2Parquet transform to proces pdf files"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "93845ab6-0338-455e-9a84-c588326f7711",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "%%time\n",
    "\n",
    "\n",
    "from dpk_docling2parquet import Docling2Parquet\n",
    "from dpk_docling2parquet import docling2parquet_contents_types\n",
    "\n",
    "STAGE = 1\n",
    "print (f\"🏃🏼 STAGE-{STAGE}: Processing input='{input_dir}' --> output='{output_docling2pq_dir}'\\n\", flush=True)\n",
    "\n",
    "Docling2Parquet(input_folder= input_dir,\n",
    "               output_folder= output_docling2pq_dir,\n",
    "               data_files_to_use=['.pdf'],\n",
    "               docling2parquet_contents_type='text/markdown').transform()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "07643473-5ca0-4392-bd1f-b140c551c0eb",
   "metadata": {},
   "source": [
    "## Step 3: Invoke PII Redactor configuration transform"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "24d847b4",
   "metadata": {},
   "outputs": [],
   "source": [
    "%%time\n",
    "\n",
    "from dpk_pii_redactor import PIIRedactor\n",
    "\n",
    "STAGE = 2\n",
    "print (f\"🏃🏼 STAGE-{STAGE}: Processing input='{output_docling2pq_dir}' --> output='{output_piiredactor_dir}'\\n\", flush=True)\n",
    "PIIRedactor(input_folder=output_docling2pq_dir,\n",
    "            output_folder= output_piiredactor_dir,\n",
    "            pii_redactor_entities = [\"PERSON\", \"EMAIL_ADDRESS\",\"ORGANIZATION\",\"PHONE_NUMBER\", \"LOCATION\",\"CRYPTO\"],\n",
    "            pii_redactor_operator = \"replace\",\n",
    "            pii_redactor_transformed_contents = \"title\").transform()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "92d77a3e-37fe-40a5-92e8-6e8ec509ac73",
   "metadata": {},
   "source": [
    "## Step 4: Display Output in a Readable Format with masked PII information"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ca040517-6c90-4f87-9eef-a618f77a1a6b",
   "metadata": {},
   "outputs": [],
   "source": [
    "data = pd.read_parquet('output/files_piiredacted/Invoice.parquet')\n",
    "print(data[\"title\"][0])\n",
    "print(data[\"detected_pii\"][0])\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "73035a54",
   "metadata": {},
   "outputs": [],
   "source": [
    "data = pd.read_parquet('output/files_piiredacted/Cryptotest.parquet')\n",
    "print(data[\"title\"][0])\n",
    "print(data[\"detected_pii\"][0])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2c69c356-df95-403f-ae6a-832d39ea0f81",
   "metadata": {},
   "source": [
    "\n",
    "### This notebook effectively demonstrates how to seamlessly apply redaction for PII entities."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
